Repository navigation
fix: registration no longer drops accounts on transient network errors - #1
Conversation
Account registration died wholesale on flaky edges: - requests silently inherited the Windows registry proxy (VPN client on 127.0.0.1) even with ip_source=direct, so every 'direct' device-flow call rode a tunnel that drops TLS at random (SSLEOFError / ProxyError) - get_zcode_jwt/get_plan had no per-request retry, so ONE escaping RequestException jumped past register_account's outer 3-attempt loop and dropped the account with only a log line generator/plan_check.py: - shared Session with trust_env=False: direct actually means direct; keep-alive across the init/authorize/callback/poll chain - _request() helper: per-request retries (3x, capped backoff) on transient RequestException for every device-flow/billing call - get_plan/get_current/preview_plans now return None/[] instead of raising on exhausted retries zcode_gateway/api_dash.py: - register_account catches per-attempt exceptions inside both retry loops so a single flake costs one attempt, not the whole registration Tests: test_claim_start / test_generator / test_accounts_dir / test_login_resilience all green (55 passed); full suite unchanged vs master. Live-verified: a mid-flow ConnectionError now retries and the account still mints + reads its plan.
There was a problem hiding this comment.
5 issues found across 2 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="zcode_gateway/api_dash.py">
<violation number="1" location="zcode_gateway/api_dash.py:882">
P3: The new catch-all handler logs only the exception type, so non-network bugs inside `get_zcode_jwt` lose the message and traceback the outer `log.exception` used to record. Include the message here, and do the same in the plan read handler below. The attempt also logs a second 'no zcodeJWT' warning right after this one.</violation>
</file>
<file name="generator/plan_check.py">
<violation number="1" location="generator/plan_check.py:118">
P2: Retrying POSTs on any RequestException can resend state-changing requests after the server already handled them. Limit the retry loop to connect-level failures for non-idempotent calls, or pass a `retry_post=False` flag for claim and event POSTs.</violation>
<violation number="2" location="generator/plan_check.py:118">
P2: `_request` does not retry the documented `502-with-empty-body` response because it returns every HTTP response without checking its status. Retry transient 5xx statuses such as 502/503/504 before returning the response.</violation>
<violation number="3" location="generator/plan_check.py:119">
P2: `get_plan` now swallows balance-request failures and returns None. `main.py` relies on an escaping exception to retry the plan check direct, so a proxy failure during the balance read now records the account as not starter with no retry. Let the exception propagate, or keep the direct-retry path.</violation>
<violation number="4" location="generator/plan_check.py:262">
P2: One burst of three failed polls raises out of `get_zcode_jwt` and discards the flow. Catch `requests.RequestException` in the poll loop, log it, and `continue` so the remaining polls can still succeed.</violation>
</file>
Reply to a comment to ask cubic a question or push back. It learns from your replies.
View guided diff | Re-trigger cubic
| last = None | ||
| for i in range(1, tries + 1): | ||
| try: | ||
| return _session.request(method, url, proxies=_proxies(), **kw) |
There was a problem hiding this comment.
P2: Retrying POSTs on any RequestException can resend state-changing requests after the server already handled them. Limit the retry loop to connect-level failures for non-idempotent calls, or pass a retry_post=False flag for claim and event POSTs.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At generator/plan_check.py, line 118:
<comment>Retrying POSTs on any RequestException can resend state-changing requests after the server already handled them. Limit the retry loop to connect-level failures for non-idempotent calls, or pass a `retry_post=False` flag for claim and event POSTs.</comment>
<file context>
@@ -84,6 +84,47 @@ def _proxies():
+ last = None
+ for i in range(1, tries + 1):
+ try:
+ return _session.request(method, url, proxies=_proxies(), **kw)
+ except requests.RequestException as e:
+ last = e
</file context>
| for _ in range(15): | ||
| r = requests.get(POLL + flow_id, headers=build_headers(poll_token, include_api_key=False), | ||
| timeout=30, verify=_VERIFY, proxies=_proxies()) | ||
| r = _request("GET", POLL + flow_id, headers=build_headers(poll_token, include_api_key=False), |
There was a problem hiding this comment.
P2: One burst of three failed polls raises out of get_zcode_jwt and discards the flow. Catch requests.RequestException in the poll loop, log it, and continue so the remaining polls can still succeed.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At generator/plan_check.py, line 262:
<comment>One burst of three failed polls raises out of `get_zcode_jwt` and discards the flow. Catch `requests.RequestException` in the poll loop, log it, and `continue` so the remaining polls can still succeed.</comment>
<file context>
@@ -213,11 +255,12 @@ def get_zcode_jwt(chat_token, log=print):
for _ in range(15):
- r = requests.get(POLL + flow_id, headers=build_headers(poll_token, include_api_key=False),
- timeout=30, verify=_VERIFY, proxies=_proxies())
+ r = _request("GET", POLL + flow_id, headers=build_headers(poll_token, include_api_key=False),
+ timeout=30, verify=_VERIFY, log=log)
try:
</file context>
| for i in range(1, tries + 1): | ||
| try: | ||
| return _session.request(method, url, proxies=_proxies(), **kw) | ||
| except requests.RequestException as e: |
There was a problem hiding this comment.
P2: get_plan now swallows balance-request failures and returns None. main.py relies on an escaping exception to retry the plan check direct, so a proxy failure during the balance read now records the account as not starter with no retry. Let the exception propagate, or keep the direct-retry path.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At generator/plan_check.py, line 119:
<comment>`get_plan` now swallows balance-request failures and returns None. `main.py` relies on an escaping exception to retry the plan check direct, so a proxy failure during the balance read now records the account as not starter with no retry. Let the exception propagate, or keep the direct-retry path.</comment>
<file context>
@@ -84,6 +84,47 @@ def _proxies():
+ for i in range(1, tries + 1):
+ try:
+ return _session.request(method, url, proxies=_proxies(), **kw)
+ except requests.RequestException as e:
+ last = e
+ if i < tries:
</file context>
| last = None | ||
| for i in range(1, tries + 1): | ||
| try: | ||
| return _session.request(method, url, proxies=_proxies(), **kw) |
There was a problem hiding this comment.
P2: _request does not retry the documented 502-with-empty-body response because it returns every HTTP response without checking its status. Retry transient 5xx statuses such as 502/503/504 before returning the response.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At generator/plan_check.py, line 118:
<comment>`_request` does not retry the documented `502-with-empty-body` response because it returns every HTTP response without checking its status. Retry transient 5xx statuses such as 502/503/504 before returning the response.</comment>
<file context>
@@ -84,6 +84,47 @@ def _proxies():
+ last = None
+ for i in range(1, tries + 1):
+ try:
+ return _session.request(method, url, proxies=_proxies(), **kw)
+ except requests.RequestException as e:
+ last = e
</file context>
| return _session.request(method, url, proxies=_proxies(), **kw) | |
| response = _session.request(method, url, proxies=_proxies(), **kw) | |
| if response.status_code in (502, 503, 504) and i < tries: | |
| log(f" plan: {method.lower()} {short} failed " | |
| f"(HTTP {response.status_code}), retry {i}/{tries - 1}") | |
| time.sleep(min(backoff * i, 6)) | |
| continue | |
| return response |
| log.warning("generator: zcodeJWT mint error for %s " | ||
| "(attempt %d/3): %s", email, attempt, | ||
| type(exc).__name__) |
There was a problem hiding this comment.
P3: The new catch-all handler logs only the exception type, so non-network bugs inside get_zcode_jwt lose the message and traceback the outer log.exception used to record. Include the message here, and do the same in the plan read handler below. The attempt also logs a second 'no zcodeJWT' warning right after this one.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At zcode_gateway/api_dash.py, line 882:
<comment>The new catch-all handler logs only the exception type, so non-network bugs inside `get_zcode_jwt` lose the message and traceback the outer `log.exception` used to record. Include the message here, and do the same in the plan read handler below. The attempt also logs a second 'no zcodeJWT' warning right after this one.</comment>
<file context>
@@ -873,7 +873,16 @@ def register_account(info: dict):
+ # transient network failure: cost ONE attempt, not all
+ # three (an escaping exception used to jump straight
+ # to the outer handler and drop the account).
+ log.warning("generator: zcodeJWT mint error for %s "
+ "(attempt %d/3): %s", email, attempt,
+ type(exc).__name__)
</file context>
| log.warning("generator: zcodeJWT mint error for %s " | |
| "(attempt %d/3): %s", email, attempt, | |
| type(exc).__name__) | |
| log.warning("generator: zcodeJWT mint error for %s " | |
| "(attempt %d/3): %s: %s", email, attempt, | |
| type(exc).__name__, exc) |
Problem
Generated accounts were silently never added to the pool. Worker logs always ended with \1 fully created, but the gateway log showed the real story:
\
ERROR zcode_gateway.dash: generator: pool registration failed for
requests.exceptions.SSLError: ... UNEXPECTED_EOF_WHILE_READING
\\
Two stacked issues:
equests\ with \proxies=None\ still inherits the IE/registry proxy (e.g. a VPN client on \127.0.0.1:10808) via \ rust_env\ — even when the generator config says \ip_source: direct. Every device-flow call rode that tunnel, which drops TLS connections at random.
egister_account's outer 3-attempt loop, dropped the account, and left only a log line. (The catch-up scan only runs at startup, so nothing rescued it.)
Fix
*\generator/plan_check.py*
equests.Session\ with \ rust_env = False: direct actually means direct; keep-alive across the init/authorize/callback/poll chain. \ZAI_PROXY\ still wins when explicitly set.
*\zcode_gateway/api_dash.py*
egister_account\ now catches per-attempt exceptions inside both retry loops, so a single network flake costs one attempt instead of the whole registration.
Evidence
Notes